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ABSTRACT 

This policy brief reviews the evolution of comprehensive 
school reform (CSR) and the development of CSR models, describes the AEL-CREP 
{Center for Research in Educational Policy) research project and its 
findings, and discusses policy implications. Over the past 2 decades, CSR has 
evolved from theory to pilot project to mandate to widespread implementation. 
Its evolution was catalyzed by reports, such as the 1983 report "A Nation at 
Risk." The AEL-CREP research project was undertaken to add to the body of 
empirical evidence on the effectiveness of CSR models. Twelve CSR schools 
were studied and compared with matching non-CSR control schools that shared 
certain characteristics. Data were collected annually. Findings showed that 
achievement gains became evident in the third year of implementation; that 
emerging differences in instructional practices seemed related to selected 
CSR models; and that different models evidenced different aspects of 
instruction, classroom practice, and teacher-student-knowledge relationships. 
This brief strongly recommends that school and policy leaders manage school 
and community expectations from the outset so that reforms are given 
sufficient time to produce positive changes, considering that student 
achievement gains were not evident until the third year of implementation. 
(Contains 16 references.) (RT) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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n 1999, AEL and the Center for Research on Educational Policy 
(CREP) at the University of Memphis initiated a three-year 
research project to examine specific school-level implementa- 
tions of comprehensive school reform (CSR). The study involved 12 of 
the 140 schools within AETs service region (Kentucky, Tennessee, 
Virginia, and West Virginia) that received funding in the initial round 
of the Comprehensive School Reform Demonstration (CSRD) program 
and 12 matched schools not engaged in comprehensive reforms. Re- 
searchers compared the CSR schools and the control schools to learn 
about (1) the effects of comprehensive school reform on school im- 
provement and (2) the differences comprehensive school reform models 
create in schools over time. 

This Policy Brief reviews the evolution of comprehensive school 
reform and the development of CSR models, describes the AEL-CREP 
research project and its findings, and draws on scientific research to 
frame policy implications. 

The Evolution of Comprehensive School Reform 

Over the past two decades, comprehensive school reform has 
evolved from theory to pilot to mandate to widespread implementation. 
It evolved, in large part, as a response to reports such as^ Nation at 
Risk (The National Commission on Excellence in Education, 1983), 
which stimulated a variety of efforts by many individuals and organiza- 
tions to make education more effective for more students. For example, 
some states raised graduation requirements, increased teacher training 
requirements, and launched efforts to assess and improve the quality of 
their schools. Universities and research centers developed programs to 
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improve various aspects of education. Communi- 
ties called for more local control of schools. Profes- 
sional associations of educators began identifying 
the necessary skills and knowledge appropriate to 
each discipline. Many of these efforts contributed 
to the emergence of standards-based education. 
They also sparked interest in determining whether 
comprehensive, or whole-school, reform might be 
more effective than what some called “piecemeal’' 
approaches. 

Until just a few years ago, most school reform 
efforts focused on individual subject matter areas 
(e.g., mathematics, science, reading) rather than 
encouraging educators to create change throughout 
a school’s system by looking across curricular areas 
and considering the effects of management, cul- 
ture, staff development, and community involve- 
ment. Several universities and research centers, 
however, had been developing comprehensive 
school reform models and testing these models in 
selected schools. 

The Obey-Porter Comprehensive School 
Reform Demonstration (CSRJD) legislation (Pub. 

L. No. 105-78, 1998) presented an opportunity for 
hundreds of schools across the nation to try com- 
prehensive reform models. The CSRD program 
provided a minimum of $50,000 per year in 
demonstration grants to schools implementing 
such reforms. The legislation was intended to 
stimulate schoolwide change, covering virtually all 
aspects of school operations. By providing financial 
incentives, it assisted schools in need of improve- 
ment, particularly Tide I schools, in implementing 
reforms designed to help all children meet chal- 
lenging state content and performance goals. The 
National Clearinghouse for Comprehensive School 
Reform (NCCSR) uses the following points to 
characterize CSR:* 

• a systematic approach to schoolwide improve- 
ment that incorporates every aspect of a 

*For an in-depth discussion of CSR definitions, visit 

NCCSR at www.goodschools.gwu.edu/about_csr/ 
index.html. 



school — from curriculum and instruction to 
school management 

® a program and a process that is designed to 
enable all students to meet challenging aca- 
demic content and performance goals 
® a framework for using research to move from 
multiple, fragmented educational programs to 
a unified plan with a single focus — academic 
achievement 

® a product of the long-term, collaborative 

efforts of school staff, parents, and district staff 

The commitment of Congress and the U.S. 
Department of Education to improving schools 
through comprehensive school reforms is under- 
scored by CSR’s incorporation into the No Child 
Left Behind Act of 2001 (NCLB) (Pub. L. No. 
107-1 10, 2002) as a formula grant program, with 
funding levels based on districts’ Title I allocations. 
As was the case with CSRD funds, schools can use 
CSR funds to support reform efforts that are 
“homegrown” (not externally developed) or re- 
forms that incorporate one or more externally 
developed, targeted, non-schoolwide interventions 
to address 9 (now 11) essential components of 
comprehensive school reform: 

1. proven methods and strategies based on scien- 
tifically based research* 

2. comprehensive design 

3. professional development 

4. measurable goals and benchmarks 

5. support within the school 

6. support for principals and teachers** 

7. parental and community involvement 

8. external technical support and assistance 

9. annual evaluation 

10. coordination of resources 

1 1. strategies that improve academic achievement** 

As NCLB is rolled out, schools must meet even 
more rigorous standards, as they will be required to 
show consistent test score increases, or Adequate 

as amended in 2002 
** added in 2002 







Yearly Progress (AYP), across curriculum areas and 
population subgroups. To comply with NCLB, 
districts' school improvement efforts must employ 
strategies developed from scientifically based 
research. This need is prompting schools to search 
for solid information on successful reform efforts 
that can be replicated within their school commu- 
nities — a type of information that is in short 
supply. 

Existing Evidence of Comprehensive School 
Reform’s Effectiveness 

Although CSRD funding was predicated on 
the use of research-based models, research on the 
effectiveness of CSR models lags far behind the 
interest and need (American Institutes for Re- 
search, 1999; Nunnery, 1998; Borman, Hewes, 
Overman, &C Brown, 2002). To date, five major 
reviews of comprehensive school reform models 
have been conducted. Generally, the following 
factors have been identified as affecting the success 
or failure of externally developed comprehensive 
reforms: (1) Schools that implement their selected 
models with the greatest fidelity are more likely to 
see positive effects; (2) Models that are more 
clearly defined tend to be implemented with 
greater fidelity and, in turn, have stronger effects 
on teaching and learning; (3) Well-implemented 
reforms usually have strong professional develop- 
ment components, including follow-up support 
that addresses specific classroom-level problems in 
implementing change; and (4) Teachers and ad- 
ministrators must support or even coconstruct the 
selected reform design (Borman, et al. 2002). 

One of the first and most often cited research 
reviews was conducted by the American Institutes 
for Research (AIR), which reviewed the claims for 
24 comprehensive school reform models. Of these, 
only 3 had evidence that met the AIR criteria for 
“strong evidence of positive effects on student 
achievement," while 5 were determined to have 
“promising evidence." Eight of the programs had 
no evidence of research on the student achieve- 



ment effects, while 2 models had weak, mixed, or 
no effects on student achievement. The remaining 
6 were found to have marginal evidence of positive 
effects on student achievement (AIR, 1999). 

In 2002, a comprehensive synthesis of existing 
research on 29 reform models' effectiveness yielded 
similar results. Geoffrey Borman and associates at 
the University of Wisconsin-Madison applied a 
rigorous methodology of research review called 
meta-analysis to categorize the models according to 
levels of effectiveness in raising student achieve- 
ment. Researchers classified 3 models as having the 
strongest evidence of effectiveness, 3 as showing 
highly promising evidence of effectiveness, 6 as 
showing promising evidence, and 17 as “in need of 
more research" (Borman, et al., 2002). The re- 
searchers concluded that the overall effects of 
comprehensive school reform appear promising, 
but cautioned that determining the specific effects 
of particular models requires careful, examination 
of how they are evaluated. 

The AELtCREP CSRD Research Project 

To enhance the body of empirical evidence on 
the effectiveness of CSR models, AEL and the 
Center for Research in Educational Policy (CREP) 
at the University of Memphis designed a research 
study that matched CSRD schools with control 
schools and tracked them over three years. The 
study focused on five specific models of compre- 
hensive school reform. While these models may be 
considered representative of the research and 
education communities' definitions of CSR, the 
reader should keep in mind that this research 
discusses schools benefiting from CSRD grants and 
the schools' development or adoption and imple- 
mentation of models in reaction to legislative 
requirements.* 

* Consult the set of three reports (one for each year of 
the study) for detailed information. The Effects of Imple- 
menting Comprehensive School Reform Models in 12 El- 
ementary Schools, by Allan Sterbinsky, Steven M. Ross, and 
Doris Redfield, can be obtained from AEL, P.O. Box 
1348, Charleston. WV 25325-1348 or 800-624-9120. 
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How the Study Was Conducted 

In the four-state region served by the Regional 
Educational Laboratory at AEL — Kentucky, 
Tennessee, Virginia, and West Virginia — approxi- 
mately 140 schools received CSR funding in the 
initial round of CSRD grants, with each receiving 
a minimum of $50,000 per year for three years. A 
variety of demographic criteria were’ applied to 
select the 12 schools that participated in the study. 
Each had adopted one of five comprehensive 
reform models: Different Ways of Knowing 
(DWoK), Success For All (SFA), Core Knowledge 
(CK), Direct Instruction (DI), or Balanced Early 
Literacy Initiative (BELI). These reform models 
emphasize different aspects of education, as de- 
scribed in the sidebar on page 7. 

Each study school was matched with a control 
school with which it shared certain characteristics 
(size, student demography, and the like) but which 
had not chosen to implement a whole-school 
reform model. By analyzing implementation and 
outcome data collected from the study schools and 
the control schools, researchers set out to identify 
the degree to which the reforms brought about 
change in student outcomes, teachers’ practice, and 
teachers’ perceptions that characteristics of the 
school — e.g., order, leadership, support — had 
changed. Recognizing that important changes 
were unlikely to be realized in the short term, the 
study team committed to a three-year study of the 
the models’ implementation. 

Data were collected from the schools using a 
variety of instruments that tested students’ reading 
ability, observed teachers in their classrooms, and 
surveyed and interviewed teachers and staff about 
their perceptions of their own schools and situa- 
tions. The data were collected annually so change 
could be tracked over both the short and the long 
term. The researchers posed two questions: 

1. What differences in classroom practices, school 

climate, and reading achievement exist between 



CSR and control schools from 1999*2002 after 
the effects of location (rural and urban) have 
been controlled? 

2. What differences exist among CSR models in 
terms of classroom practices, school climate, 
and student achievement after at least three 
years of implementation? 

Terminology Used in tiiie Study 

In developing a series of survey instruments 
used in the study, the University of Memphis 
employed several terms that are useful in describ- 
ing aspects of school climate and attributes of 
school reform models. For example, the School 
Climate Inventory (SCI), an instrument developed 
at CREP in 1989 (Butler & Alberg, 1991), has 
been used for school-based improvement planning 
in schools and districts in numerous states, and has 
proven to be especially valuable for monitoring and 
managing change initiatives. The SCI assesses 
seven dimensions that are logically linked with 
factors associated with effective school organiza- 
tional climates: 

® order: the extent to which the environment is 
ordered and appropriate student behaviors are 
present 

® leadership: the extent to which the administra- 
tion provides instructional leadership 
® environment: the extent to which a positive 
learning environments exist 
® involvement: the extent to which parents and 
the community are involved in the school 
• instruction: the extent to which the instruc- 
tional program is well developed and imple- 
mented 

® expectations: the extent to which students are 
expected to learn and be responsible 
® collaboration: the extent to which the admin- 
istration, faculty, and students cooperate and 
participate in problem solving 

Another instrument, the Comprehensive 
School Reform Teacher Questionnaire (CSRTQ), 
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is a modification of an instrument originally 
developed in 1996 for the formative evaluation of 
Memphis City Schools' implementation of New 
American Schools restructuring designs (Ross et 
ah, 1997). For lise in schools undergoing compre- 
hensive change, the instrument has been modified 
to include generally accepted school reform criteria 
such as the 1 1 CSR criteria. The CSRTQ exam- 
ines four dimensions: 

® support: the adequacy and value of support 
provided to the school 

• capacity:, the presence of available material, 
time, and human resources 

• pedagogy: changes in classroom practices 

• outcomes: impact on student achievement and 
enthusiasm, parent/community involvement, 
and collaboration 

The final section of the CSRTQ includes 
implementation benchmarks designed to rate the 
level of model implementation in six areas: cur- 
riculum, instruction, parent/community involve- 
ment, allocation of resources, professional develop- 
ment, and evaluation.* 

Findings 

Analysis of the. data collected through class- , 
room observation, interviews with teachers and 
principals, and surveys about perceptions revealed 
the following: 

Achievement gains became evident in the 
third year of implementation. Perhaps the most 
compelling finding was that on some measures, 
reading achievement among the students in the 
CSR schools was notably higher — to a statistically 
significant level — than it was for students in the 
non-CSR schools. This finding is especially impor- 
tant because the significantly higher achievement 
did not appear until the third year of the study. 
Therefore, it can reasonably be inferred that if the 

For more information on these instruments, contact the 
Center for Research in Educational Policy at the Univer- 
sity of Memphis, 325 Browning Hall, Memphis, TN 
38111-9861, 901-678-2310. 
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CSR schools had not continued the reform efforts 
beyond year two, they would not have seen the 
student achievement gains that they did. This 
finding reinforces the proposition put forward by 
many who study school reform that providing 
enough time for the effects to be felt is crucial. 

Faculty and staff at schools implementing 
comprehensive refprms perceived some elements 
of school climate as being more positive. Differ- 
ences associated with school climate were noted 
between CSR and control schools. CSR schools' 
faculty and staff were more likely to be positive in. 
their perceptions of order, instruction, and expec- 
tations than were staff in control schools. In urban 
CSR schools as compared to urban control schools, 
teachers were significantly more positive about 
leadership and expectations. 

Emerging differences in instructional prac- 
tices seemed related to selected comprehensive 
school reform models. Moreover, there were 
differences in instructional practices between the 
CSR schools and the control schools and between 
urban and rural schools. After three years of 
implementation of the reform models, teachers at 
CSR schools were more likely to use direct instruc- 
tion and ability grouping, higher-order questions, 
sustained writing, and student discussion than 
were teachers at the control schools. These teach- 
ing practices are all emphasized (to a greater or 
lesser degree) by the particular reform models 
implemented. 

Different models evidenced different aspects 
of instruction, classroom practice, and teacher- 
student-knowledge relationships. Because the 
models emphasized different aspects of instruction, 
classroom practice, and teacher-student-knowledge 
relationships, it would not be surprising to find 
differences that could be traced to specific models. 
When researchers examined differences among four 
of the five models (DWoK was implemented in 
only one school in the study, so it could not be 
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included in the model analysis), what they found 
among study schools supported this expectation: 

® Balanced Early Literacy Initiative (BELI) 

teachers used parent/community involvement 
and individual tutoring significantly more 
often than did other teachers, and had higher 
perceptions of expectations and order. 

® Core Knowledge (CK) teachers used more 
direct instruction, work centers, higher-level = 
questioning, hands-on learning, and sustained 
writing than did other teachers, although the 
CK model does not require the use of specific 
pedagogic practices. At CK schools, teachers’ 
perceptions of expectations, order, pedagogy, 
and collaboration were significantly higher, but 
their perceptions of support were the same as 
other models. 

® Direct Instruction (DI) schools reported lower 
levels of implementation of the reform than 
did schools using other models. DI teachers 
also reported lower perceived levels of peda- 
gogy, order, involvement, expectations, and 
outcomes than teachers using other models. 

DI schools were observed to use more ability 
groups than schools using other models. 

® Success For All (SFA) schools used more 

student-centered classroom practices than did 
other schools, and SFA teacher perceptions of 
order were lower. SFA teachers reported higher 
levels of capacity and pedagogy, but their 
perceptions of support were the same as in 
other models. 

It should be remembered that each CSR school 
chose a reform model it believed would best suit its 
particular needs and circumstances. Not surpris- 
ingly, each school demonstrated specific aspects of 
its chosen model more than did schools using 
other models. 

Conclusions and Issues for Further Study 

The improvements made through the CSR 

|(g IMmslixs? 



implementations evaluated in this project, with 
respect to both student achievement and instruc- 
tional practice, are visible and supported by the 
findings described here. The fact that student 
achievement gains were not evident until the third 
year of implementation (other studies have shown 
it might take as much as four or five years) under- 
scores the need for school and policy leaders to 
manage school and community expectations from 
the outset so that reforms are given sufficient time 
to effect positive changes. 

NCLB legislation seems to encourage schools 
to broaden the scope of comprehensive school 
reforms in a different way. It de-emphasizes reli- 
ance on reform models and encourages schools to 
focus on strategies that address the 1 1 components 
named in the legislation, with a specific focus on 
raising student achievement. Schools may continue 
to use commercial or “prepackaged” reform models 
as part of their overall strategy for school improve- 
ment, but the shift in the legislation’s language 
might encourage more schools to develop “home- 
grown” reform models. One model in the AEL- 
CREP study, BELI, is a locally developed model 
that centers on a reading program but also incor- 
porates other strategies to meet all CSR criteria. 
Additional research on the demands and effective- 
ness of such homegrown models could help schools 
determine the best course of action for their 
particular circumstances. 

The NCLB legislation also creates a need for 
research that focuses on strategies rather than 
specific models, as schools and districts seek clear 
information on the relative successes of different 
CSR intervention methods. Perhaps equally impor- 
tant will be studies that specify the school settings 
and parameters within which specific strategies are 
effective. 

The AEL-CREP study opens the door to 
another CSR question that the education research 
community will grapple with in the coming 
months and years of CSR implementation: Once a 
strategy proves to be effective, what policies and 
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Success For All (SFA) was developed at the 
Johns Hopkins University and uses preventive 
and early intervention techniques to ensure 
that every child is a successful reader and 
student. SFA has nine components: reading 
curriculum including at least 90 minutes of 
daily instruction with homogeneous groups 
across age lines, eight- week assessments, 
tutoring, early learning program for 
prekindergarten and kindergarten students, 
cooperative learning, a family support team, a 
facilitator, staff support teams, and training 
and technical assistance on specific topics. 
(Success For All Foundation, 2000) 

Different Ways of Knowing (DWoK) was 
created by the Galef Institute and is based on 
five philosophical components: positive expec- 
tations for the academic and social achieve- 
ment of all children, thematically integrated 
instruction across disciplines, active student 
participation, early intervention, and parent 
participation in the classroom and at home. 
The curriculum integrates history and science 
with literature, language arts, math, science, 
and the arts. The arts focus is used as a path- 
way to all disciplines. (The Galef Institute, 
2002 ) 

Balanced Early Literacy Initiative (BELI) 
is a locally developed model that is based on 
the components of the Early Literacy Initiative 



from Ohio State University. It is an early 
intervention program (K-4) designed to 
provide all students with the experiences and 
instruction necessary to help children become 
good readers at an early age. Program compo- 
nents include: Rreading aloud, shared reading, 
guided reading, independent reading, shared 
writing, interactive writing, guided writing, 
and independent writing. In addition to these 
components. Success By Seven (one-on-one 
structured lessons) is provided to all at-risk 
first graders. 

Direct Instruction (DI) was developed by 
Siegfried Engelmann at the University of 
Illinois in the 1960’s and uses tightly scripted 
and interactive lesson plans with small, homo- 
geneous groups of children based on their 
performance levels. The curriculum covers 
reading, language arts, and math,; and uses 
frequent assessments are used. (SRA/McGraw- 
Hill, 2002) 

Core Knowledge (CK) is based on the 
premise that all citizens need a common 
knowledge base (CK) to function well in 
society. Schools provide this shared knowledge 
to students by using a curriculum framework 
that specifies important knowledge in language 
arts, history, geography, math, science, and the 
fine arts that is sequenced within and between 
grades. (Core Knowledge Foundation, 2002) 



practices must be put in place to sustain improve- 
ments and create lasting success? An analysis of 
reform efforts in Memphis City Schools between 
1995 and 2001 suggests that successful efforts are 
the result of a delicate balance of administrative 
and teaching staffs’ readiness for change, attention 
to local issues within the school, buy-in from all 
stakeholders, professional development, and focus 
on the most pressing needs of the school (Ross, 
2001). Long-term studies can yield the kind of 



scientifically based data schools and policymakers 
need to successfully sustain school improvements. 

Yet another issue that needs further research is 
the correlation between school climate and aca- 
demic achievement/Adequate Yearly Progress. 
Because school climate is an important component 
of most CSR models, research on this factor of 
CSR could yield important findings for school 
leaders and policymakers as they create or support 
strategies for school improvement. 
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